Advanced Bloom Filter Based Algorithms for Efficient Approximate Data De-Duplication in Streams

Data intensive applications and computing has emerged as a central area of modern research with the explosion of data stored world-wide. Applications involving telecommunication call data records, web pages, online transactions, medical records, stock markets, climate warning systems, etc., necessitate efficient management and processing of such massively exponential amount of data from diverse sources. Duplicate detection and removal of redundancy from such multibillion datasets helps in resource and compute efficiency for downstream processing. De-duplication or Intelligent Compression in streaming scenarios for approximate identification and elimination of duplicates from such unbounded data stream is a greater challenge given the real-time nature of data arrival. Stable Bloom Filters (SBF) addresses this problem to a certain extent. However, SBF suffers from a high false negative rate and slow convergence rate, thereby rendering it inefficient for applications with low false negative rate tolerances. ∗This work was completed at IBM Research, India. Email addresses: [email protected] (Suman K. Bera), [email protected] (Sourav Dutta), [email protected] (Ankur Narang), [email protected] (Souvik Bhattacherjee) Preprint submitted to Information Systems, Elsevier January 9, 2014 ar X iv :1 21 2. 39 64 v1 [ cs .I R ] 1 7 D ec 2 01 2 In this work, we present several novel algorithms for the problem of approximate detection of duplicates in data streams. We propose the Reservoir Sampling based Bloom Filter (RSBF) combining the working principle of reservoir sampling and Bloom Filters. We also present variants of the novel Biased Sampling based Bloom Filter (BSBF) based on biased sampling concepts. Using different updation and biasing mechanisms we propose variants of the same model enabling the data structure to adapt to various input scenarios. We also propose a randomized load balanced variant of the sampling Bloom Filter approach to efficiently tackle the duplicate detection. In this work, we thus provide a generic framework for de-duplication using Bloom Filters. Using detailed theoretical analysis we prove analytical bounds on the false positive rate, false negative rate and convergence rate of the proposed structures. We exhibit that our models clearly outperform the existing methods. We also demonstrate empirical analysis of the structures using real-world datasets (3 million records) and also with synthetic datasets (1 billion records) capturing various input distributions.